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Abstract 

In this paper, we use Conley index theory to examine the Poincare index 
of an isolated invariant set. We obtain some limiting conditions on a critical 
point of a planar vector field to be an isolated invariant set. As a result we 
show the existence of infinitely many homoclinic orbits for a critical point 
with the Poincare index greater than one. 
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1 Introduction 

The Conley index has proved to be a useful tool in the investigation of qualitative 
properties of dynamical systems. It has generalized Morse theory for an isolated 
invariant set of a continuous flow on a locally compact metric space 0, §]. For this 
reason, Conley index is known as a generalization of Morse theory. In || and [[3], 
Conley and Zehender used this index to show the existence of periodic solutions for 
Hamiltonian systems. This was a landmark in the proof of the Arnold conjecture 
on the existence of periodic orbits of Hamiltonian systems on symplectic manifolds. 
Conley index theory has also some applications in the existence of solutions for a 
class of differential equations. (See |L5], [L7[ 0] and references therein.) 
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In Conley index theory, we deal with pairs of closed sets called index pair for an 
isolated invariant set /. The homotopy type of these index pairs is independent of 
the index pair chosen, which is called the Conley index of / and denoted by h(I). 
This paper concerns the relation between Conley index theory and the Poincare-Hopf 
theorem ||. We define the Poincare index of an isolated invariant set / to be x{h{I))- 
This definition coincides with the classical Poincare index when the invariant set is 
a single point. We use some topological properties of the Conley index to obtain 
restrictions on the Poincare index of isolated invariant sets in dimension two. It is 
well-known that on a two-dimension manifold M, the Poincare index of an isolated 
critical point of a gradient vector field is not greater than one. Here we show that a 
critical point x with ind(x) > 1 cannot be an isolated invariant set. This concludes 
the existence of infinitely many homoclinic orbits for such a critical point. 

We first present some basic results from Conley index theory. Then we define 
the concept of continuation and provide a new proof for the results of || based 
on continuation to gradient [|l|, Finally we apply these results to show the 
existence of infinitely many homoclinic orbits in dimension two. 

2 Conley Index 

Let ip 1 be a C 1 -flow on a smooth manifold M. A subset I C M is called an isolated 
invariant set if it is the maximal invariant set in some compact neighborhood of 
itself. Such neighborhood is called an isolating neighborhood. 

Definition. A closed pair (JV, L) is called an index pair for I if 

1. N — L is an isolating neighborhood for I. 

2. L is positively invariant relative to N, i.e., if x G L, t > 0, (pl°d(x) C N, then 
ip^(x) c L. 

3. L is the exit set of N, i.e., if x G N, t G M + and ip t {x) N, then there is a 
t! G [0,t] such that y/(x) G L. 
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In [|1], [3], [T6|, [TJJ it has been shown that every isolated invariant set / admits 
an index pair (N,L) and the homotopy type of (N/L, [L]) is independent of the 
index pair chosen. We denote the homotopy type of (N/L, [L]) by h(I) and call it 
the Conley index of I. The homology Conley index of I is defined by CH*(I) = 
H,(N/L, [L]). 

c 2 

Example 2.1. Let x G M be a nondegenerate critical point for / : M — > R. 
Then {x} is an isolated invariant set for —V/ and by Morse Lemma [|j], it is easy 
to show that h({p}) is a pointed k-sphere where k is the number of positive eigen- 
values of Hessian matrix / at p. Therefore the Conley index can be considered as a 
generalization of Morse index. 

It is not true that H*(N, L) ^ H*(N/L, [L]) for every index pair (N, L). In M, 



Salamon introduced a class of index pairs for which the above isomorphism holds. 

Definition. An index pair (N, L) is called regular if the exit time map 
r+ : N—> [0,+oo], r + (x) = 



sup{t\^ t] (x) CN-L} if x e N - L, 
if x e L, 



is continuous. ( See |16| for more details about regular index pairs.) For every 
regular index pair (N, L), we define the induced semi- flow on iV by 

(f \ : JV X R+ >N, <p\(x) = ^rmn^r+ix)}^ 

Proposition 2.2. If (N, L) be a regular index pair for a continuous flow (p*, then 
L is a neighborhood deformation retract in N. In particular, the natural map 
7i : N — > N/L induces an isomorphism H*(N, L) = H*(N/L, [L]). 

Proof. Consider the induced semi-flow cp§ on and the neig hborhoodf/ := t^[0,1] 
of L. Now ^|i/x[o,i] gives the desired deformation retraction. □ 



In ||14|| , Robbin and Salamon proved that every isolated invariant set admits 
a regular index pair which is stable under perturbation. They first showed the 
existence of a smooth Liapunov function on a neighborhood of the isolated invariant 
set. 
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Theorem 2.3. Let N be an isolating neighborhood of /. Then there is a neigh- 
borhood U of N and a smooth function / : U — >WL satisfying 

(i) f(x) = for all x e I. 

(ii) 4|t=o/(y*(^)) < for all x G N — I. (/ decreases along orbits in U — I.) 

Then they used this Liapunov function to construct a triple (N, L~ , L + ), such 
that (N, L + ) is a regular index pair for / with respect to the forward flow and (N, L~) 
is a regular index pair for / with respect to the reverse flow. Furthermore L + and 
L~ can be chosen to be (n-l)-manifolds with boundary, so that N is a manifold with 
corners with those corners contained in L~ PI L + and N = L~ U L + . We call such 
a triple (N, L~ , L + ) as a regular index triple for / in M. The Conley indices of / 
related by the forward and reverse flow are represented by h + (I) and h~(I). If M 
is orientable in a neighborhood of I, the indices for the forward and reverse flows 
are related by Poincare-Lefschetz duality isomorphism H*(N,L + ) ~ H m ~*(N, L~) 
where m = dimM . (See || H, |18|.) If we consider the homology with coefficients in 
Z2, the Poincare-Lefschetz duality is valid without the assumption of orientability. 

Definition. A C M is called an attractor set if it is the cu-limit set of a compact 
neighborhood of itself. A repeller set is an attractor set for the reverse flow. 

Proposition 2.4. / is an attractor set for 99* if and only if there is an index pair 
(N, L) for I which L = 0. 

Proof. Let / be an attractor and V be a neighborhood of / such that ou(V) = I. 
Then there is a T > such that ^ T '°°\V) C int{V). If we set N := C\o< s <t f s (V) , 
then (N, 0) is an index pair for I. Now assume that (N, 0) is an index pair for /. 
According to the property (3) of the definition of index pair, we imply that iV is 
positively invariant, hence u(N) C N. Since iV is an isolating neighborhood for /, 
it follows that u(N) C I. Since I is an invariant set, we conclude that u(N) — I. □ 

Theorem 2.5. Suppose that / C M is a connected isolated invariant set. 
(i) If / is not an attractor, then H (h + (I)) = 0. 
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(ii) If I is not a repeller, then H m (h + (I); Z2) = 0. Moreover if M is orientable in 
a neighborhood of I, then H m (h + (I)) = 0. (m = dimM) 

Proof. Consider a regular index triple (N, L + , L~~) for /. We may assume that iV 
is connected (otherwise replace N by the connected component of N that contains 
I). Since / is not an attractor set, L + ^ by Proposition 2.4. Thus 

H (h+(I)) = H (N,L+) = 

Similarly we have L~ ^ and H°(h~(I)) = 0. Now by Poincare-Lefschetz duality 
H m (h+(I))~H°(h-(I)) = 0. □ 



Definition. A Morse decomposition of / is a finite collection {Mj}™ =1 of disjoint, 
nonempty isolated invariant subsets of I such that for each x G I — ([X=i ^')> there 
are 1 < i < j < n such that a(a;) G Mj and ^(x) G M,. 



Example 2.6. Consider the gradient flow of a smooth function / on a compact 
manifold M. Suppose that {x±, ■ ■ ■ ,x n } are critical points of / with f(xi) < f(xj) 
for i < j. Then {x±, ■ • • , x n } is a Morse decomposition for M. 



Theorem 2.7. Let / be an isolated invariant set with a Morse decomposition 
{Mj}™ =1 . Define the Poincare polynomial p(t, X) := Y^T=o^{X)t k that &fc(X) = 
dim(Hk(X)) is k-th Betti number of X. Then there is a polynomial Q with nonneg- 
ative coefficients such that 



Y.vit, h(Mi)) = p(t, h(I)) + (1 + t)Q(t) 



i=l 



The above theorem is known as the generalized Morse inequalities |3|, pufl . If 
we apply this result to the gradient flow of a Morse function with the Morse de- 
composition described in Example 2.6., we obtain the classical Morse inequalities 
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3 Continuation 



A parametrized flow on M is a collection of flows {(p* \ X E 1} indexed by / = [0, 1] 
such that $t(x, A) = (p^(x), A) is a C^-flow on M x I. we say S°, an invariant set for 
<Pt, and 5 1 , an invariant set for p], are related by continuation if there is an isolated 
invariant set E C M x / for $ t such that S* = S H M x {0} and S 1 = £ n M x {1}. 
The reason that continuations are interesting in Conley index theory is the following 
theorem [0, [L7|. 



Theorem 3.1. If S° and S* 1 are isolated invariant sets related by continuation, 
then h(S°) = hiS 1 ). 

f we have an isolated invariant set S and make a small perturbation of the flow, 
then the new flow will have an isolated invariant set S' near S, and by the above 
theorem h(S) = h(S'). Therefore Conley index is invariant under perturbation. In 



12, [L3| Reineck has shown that every isolated invariant set can be continued to an 



isolated invariant set in a Morse-Smale gradient flow. Since Morse-Smale flows are 
easy to deal with, we first prove our result for a Morse-Smale flow and then extend 
it to the general case by using the invariance of Conley index under continuation. 



Theorem 3.2. Let X be a smooth vector field on a Riemannian manifold M and 
let I be an isolated invariant set in (p t , the flow generated by X, with isolating 
neighborhood N. Then / can be continued to an isolated invariant set in a Morse- 
Smale gradient flow without changing the vector field on M — N. 



Definition. We define the Poincare index of an isolated invariant set I to be the 
Euler characteristic of the Conley index of /, i.e. ind p (I) := x(h(I)). 

Suppose that the flow (p* is associated with a vector field X on M. If {x} is a 
critical point of X and an isolated invariant set for ip l , then ind p (x) coincides with 
the classical definition of Poincare index of x (up to a sign). This is a special case of 
the results of || in which McCord developed the Poincare- Hopf theorem and showed 
that ind p (I) = (— l) m ^ ind(x), where the sum is taken over all critical points in 
/, ind(x) is the Poincare index of x relative to vector field X and m = dimM. We 
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provide another proof for this result based on Reneik's continuation to gradient. 

Theorem 3.3. Let I be an isolated invariant set. Then 

ind p {I) = (-l) m J2^d{x). 

In particular, if ind p (I) ^ 0, then there exists a critical point in I. 

Proof. By Theorem 3.2., / can be continued to an isolated invariant set J in a 
Morse-Smale gradient flow —V/ without changing the vector field onM-iV where 
N is an isolating neighborhood for /. Thus h(I) = h(J) by Theorem 3.1. If {yi}f =l 
is the set of critical points of / in N, then J2 xeI ind(x) = Y^i=i ind{%/i). If the 
Morse index of yi is k, then ind(y,j) = (— l) m+fe and by Example 2.1, h({yi}) = 
Let Hk denotes the number of critical points of Morse index k. Thus ^2 xeI ind(x) = 
J2k(~ l) m+ Vfc- By Example 2.6., {yi}f =1 is a Morse decomposition for J with respect 
to —V/. According to the generalized Morse inequalities (Theorem 2.11), we have 

n 

5>(-l, %*)) = P(-1, HJ)) = p(-l, h{I)). 

i=l 

Since = S fc , we get p(t, h(yi)) = t k and = p( - 1> M-0)- Therefore 

oo 

(-ir^md(x) =p(-l, fc(J)) = ^(-l) fe fo fe (M/)) = xW) = □ 
ie.r fc=o 

Proposition 3.4. Suppose that I C M is an NDR (Neighborhood Deformation 
Retract) isolated invariant set. 

(i) If / is an attractor, then ind p (I) = x(-0- 

(ii) If 7 is a repeller, then ind p (I) = (-l) m x(I). (m = dimM) 

Proof. When I is an attractor, there is an index pair (N, 0) for I by Proposition 2.4. 
Since I is an NDR, there exists a neighborhood U C N such that I is deformation 
retract of U. By the definition of index pair, N is positively invariant and cu(N) = 
I. So there is a T > such that y? T (iV) C £/. Therefore iV can be deformed 
to I and Hi(N,0) = Hi(I) for every i Thus x(M-O) = x(-0 which proves (i). 
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Notice that for a finite CW-complex, the Euler characteristic does not depend on 
the coefficients field. Since / is assume to be an NDR, it has the homotopy type of 
a finite CW-complex. If we consider the homology with coefficients in Z 2 , we obtain 
x{h + (I)) = (—l) m x(h~(I)) by the duality theorem. So if I is an NDR repeller, then 
ind p (I) = (-l) m X (I)- □ 

4 Applications 

In this section, we consider a smooth vector field on a surface M with an isolated 
critical point x. We desire to show that if ind(x) > 1, then x is accumulated by 
infinitely many homoclinic orbits. 

Lemma 4.1. Let I C M be a connected NDR isolated invariant set such that 
ind p (I) > 0. Then I is either an attractor or a repeller and ind p (I) = x(/). 

Proof. Suppose that / is neither an attractor nor a repeller. By Theorem 2.5., 
H 2 (h(I); Z 2 ) ~ H (h{I); Z 2 ) = 0. Now we conclude that 

ind p (I) = X {h(I)) 

= rank(H 2 (h(I); Z 2 )) - rank(H 1 (h(I);Z 2 )) + rank(H (h(I); Z 2 )) 
= -rank(H 1 (h(I);Z 2 )) < 0. 

Since m = 2, the proof is complete by Proposition 3.4. □ 

Theorem 4.2. Let x be a critical point for a vector field on a surface M. If 
ind(x) > 1, then there exists a homoclinic orbit in any neighborhood of x. 

Proof. We first show that {x} cannot be an isolated invariant set. Suppose the 
contrary, then according to Theorem 3.3. and the above lemma, ind(x) = x{{ x Y) = 
1 which is a contradiction. Consider a closed neighborhood V of x with no critical 
points rather than x. Let I(V) be the maximal invariant set in V. The above 
argument says that there is a point y 7^ x in I{V), hence uj(y) C I{V). Notice that 
there cannot be any cycle in V. To see this, suppose that is 7 is a cycle in V. Then 
the Poincare index of 7 must be one 0, thus there exists a critical point inside 7. 
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Since the only critical point in V is x, we get ind(x) = 1 which is a contradiction. 
Now according to the Poincare-Bendixon theorem P, [Ufl , and a(y) are critical 
points or homoclinic orbits. If neither of u)(y) and a(y) are homoclinic orbits, then 
w{y) = ac(y) = x. So there exists a homoclinic orbit in V. □ 



Proposition 4.3 Let 7 be a homoclinic orbit with no critical point inside of it. 
Then all the orbits inside 7 are homoclinic. 

Proof. Let x := ou(j) = a(x) and Q be the region surrounded by 7. Similar to 
the above argument, there is no cycles in Q and the limit sets of any orbit in Q are 
either {x} or homoclinic orbits. Since x is the only critical point in Q, it belongs to 
all limit sets. On the other hand, it is known that if one of the limit sets is not a 
critical point, then the limit sets are disjoint || [TO]. Therefore the limit sets of any 
orbit in Q must be {x}. □ 



Remark 4.4. It is well-known that if a; is a critical point of a gradient vector field, 
then ind(x) < 1. ( See |lT| for another proof.) The above theorem clearly shows 
why this result is true. 
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